ORIGINAL ARTICLE 

Occupation d Environmental Medicine 



http://dx,doi.org/10.3346/jkms,201 1,26,1 1,1409 • J Korean Med Sci 201 1 : 26: 1409-1414 



JKMS 



Validity of Self-reported Healthcare Utilization Data in the 
Community Health Survey in Korea 



Hwayoung Rim\ Hyeongsu Kim\ 
Kunsei Lee\ Sounghoon Chang\ 
IVIelbourne F Hovel! ^ Young-Taek Kim^ 
Yuna Kim^, Gilwon Kang^ Yangju Tak^ 
and Jeehye Im^ 

'Department of Preventive Medicine, Scliool of 
Medicine, Konkuk University, Seoul, Korea; ^Center 
of Belnavioral Epidemiology and Community Health, 
Graduate School of Public Health, San Diego State 
University, San Diego, CA, USA; ^Division of Chronic 
Disease Control and Prevention, Korea Centers for 
Disease Control and Prevention, Osong; "Department 
of Health Informatics and Management, College of 
Medicine, Chungbuk National University, Cheongju; 
'"Department of Emergency Medical Service, College 
of Health and Life Science, Chungju National 
University, Chungju; "^Center for Health Care 
Research, Health Insurance Review and Assessment 
Service, Seoul, Korea 

Received: 12 June 2011 
Accepted: 6 September 2011 

Address for Correspondence: 
Hyeongsu Kim, MD 

Department of Preventive Medicine, School of Medicine, Konkuk 
University, 263 Achasanro, Gwangjin-gu, Seoul 143-701, Korea 
Tel: +82.2-2030-7942, Fax: +82.2-2049-6192 
E-mail: mubul@kku.ac.kr 

This study was supported by a 2010-Grant from Korean 
Academy of Medical Sciences. KMA. 



To evaluate the sensitivity and specificity of Community Healtli Survey (CHS), we analyzed 
data from 11,217 participants aged > 19 yr, in 13 cities and counties in 2008. Three 
healthcare utilization indices (admission, outpatient visits, dental visits) as comparative 
variables and the insurance benefit claim data of the Health Insurance Review ft Assessment 
Service as the gold-standard were used. The sensitivities of admission, outpatient visits, 
and dental visits in CHS were 54.8%, 52.1%, and 61 .0%, respectively. The specificities 
were 96.4%, 85.6%, and 82.7%, respectively. This is the first study to evaluate the 
validity of nationwide health statistics resulting from questionnaire surveys and shows that 
CHS needs a lot of efforts to reflect the true health status, health behavior, and healthcare 
utilization of the population. 
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INTRODUCTION 

Identifying the current status of health indices sets the founda- 
tion for a series of processes in health programs, including selec- 
tion of program priorities, the establishment of purposes and 
goals, and the development of strategy, program implementa- 
tion, and evaluation. Efforts to identify the health status of the 
nation were initiated with the 'National Health Survey' in 1962 
and the 'National Nuttition Survey' in 1969. Currentiy the 5th 
'Korean National Health and Nutrition Examination Survey 
(KNHANES)' is in progress. As KNHANES considers only 16 
wide-area self-governing districts, this nationwide sampling 
design has failed to cover the actual current status of 253 basic 
regional self-governing districts in which health programs are 
executed (1). 

To overcome this limitation and to create health indices that 



are comparable among regions, the Community Health Survey 
(CHS) has been conducted annually since 2008 in individuals 
aged > 19 yr in 253 regions around the country (2). The CHS has 
a two-stage sampling process. The first sampling stage is the pro- 
cess of selecting a sample area (tong/ban/ri) that is a primary 
sample unit, selected according to the number of households 
in dong/eup/myeon (the smallest administtative unit) using a 
probability proportional to the sampUng method. In the second 
sampUng stage, as a process of selecting sample households, the 
number of households in the selected sample tong/ban/ri is 
identified to create a household directory, and sample house- 
holds are selected through systematic sampling methods. These 
methods are used to ensure the sample units are representative. 

To evaluate reliability of CHS, a third organization conducts a 
telephone survey in 5% of the respondents 2 weeks after the in- 
terview. This identified how well the responses match. In fact. 
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the kappa values were, on average, 0.68 (0.5 1 to 0.90) in 2008 and 
averaged 0.80 (0.70 to 0.90) in 2009 (3, 4). 

However, the validity of the CHS has not yet been assessed. 
We evaluated the accuracy of self-reported CHS obtained by anal- 
ysis of both sensitivity and specificity for some healthcare utili- 
zation. 

MATERIAL AND METHODS 
Study subjects 

The CHS of Chungbuk-do was performed on 12,449 participants 
aged > 19 yr, in 13 cities and counties in 2008. This study used 
data of 11,217 participants, excluding 1,206 participants who did 
not allow the use of their data for national statistics connected 
with other organizations (e.g., health insurance, death data) and 
26 who agreed but provided incorrect residency regjsttation num- 
bers. The CHS of Chungbuk-do was conducted from September 
1st to November 30th, 2008. 

Selection of questionnaire items for validity evaluation 

The questionnaire items regarding healthcare utilization for the 
validity test have to have sufficient responses in CHS and gold- 
standard tests to be objectively compared. This study selected 
the following three questions; 1) Have you been hospitalized in 
the past year? Yes/No (admission), 2) Have you been examined 
as an outpatient in the past 2 weeks? Yes/No (outpatient visit), 
3) Have you visited a dental clinic in the past year? Yes/No (den- 
tal visit). The actual survey question used to assess the dental 
visits was; when was the last time you visited the dentist? © within 
the last 6 months, © over 6 months-within the last year, ® over 
1 yr- within the last 3 yr, ® over 3 yr, © no visit, and ® unknown/ 
cannot remember. The response based on 'within the last year 
or not' in this study was changed by categorizing the answers 
into two Yes/No answers, those who had visited in the last year 
(®, ©) and those who had not (©-©). 

Gold-standard for validity tests 

The insurance benefit claim data of the Health Insurance Review 
& Assessment Service (HIRA) was used as the gold-standard for 
the validity test of the three questions investigated in this study. 

Merging of the two data sets 

Resident registration numbers offered by respondents were used 
to merge the CHS data and the HIRA's insurance benefit claim 
data. The two data sets were merged by HIRA and a new identi- 
fication number was assigned to the merged data. Resident reg- 
istration numbers were deleted to protect the private informa- 
tion of the participants. 

Data fabrication 

The period of the respondent's healthcare utilization in the HIRA 
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claim was within the last year for admission and dental visits, 
and within the last 2 weeks for outpatient visits, based on the 
survey date of the CHS. The respondent's healthcare utilization 
for admission or outpatient visits was defined by admission or 
outpatient visit cases that were claimed by all healthcare insti- 
tutions (including dental clinics) for insurance benefits to the 
HIRA during the assigned period. In the case of dental visits, it 
was defined with the cases (admission or outpatient visits) that 
were claimed by only dental healthcare institutions. 

Validity tests 

The sensitivity of each question is the rate of those who used a 
healthcare organization among those whose data from the HIRA 
identifies their healthcare utOization in the CHS. The specificity 
of each question is the rate of those who claimed not to use a 
healthcare organization among those whose data from the HIRA 
identifies their non-use of healthcare in the CHS. 

Data analysis 

The frequency of healthcare utilization was initially calculated 
from the two data sets. The sensitivity and specificity of the three 
questions were then assessed. Additionally, the sensitivity and 
specificity of the detailed variables were assessed to test the sig- 
nificance between the sensitivity and specificity of each variable. 
A P value < 0.05 (two tailed test) was considered to indicate sta- 
tistical significance. 

Ethics statement 

This study was reviewed and approved by the institutional re- 
view boards of Konkuk University Hospital (Approval number: 
KUH1230005). We received informed consents from all of the 
participants in the interview survey. 

RESULTS 

Respondent's healthcare utilization in HIRA and CHS 

The rates of admission and outpatient visits were 13.4% and 
36.6% in HIRA and 10.4% and 28.2% in CHS, respectively. The 
rate of dental visits was 44.7% from HIRA and 30.6% from CHS 
(Table 1). 

Sensitivity of healthcare utilization 

The sensitivities of admission, outpatient visits, and dental visits 
in CHS were 54.8%, 52.1%, and 61.0%, respectively (Table 2). No 
significant difference was evident from the sensitivity of admis- 
sions according to gender or marital status. But the values were 
signiflcantiy lower for those participants aged in their 20s (38. 1%) 
and 30s (45.3%), those with 10-12 yr (50.3%) and more tiian 13 yr 
(45.3%) of education, and those who were health insurance hold- 
ers (54.0%; P < 0.01). The sensitivity for outpatient visits was sig- 
nificantiy lower for males (47.8%), those aged in their 20s (30.0%), 
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Table 1. Healthcare utilization of study popuiation in HIRA and CHS 



Parameters 






Admission 


Outpatient visit 


Dentai visit 


Yes 


No 


Yes 


No 


Yes 


No 




HiRA 


11,217(100.0) 


1,500 (13.4) 


9,706 (86.6) 


4,096 (36.6) 


7,105 (63.4) 


A AAA / A A ~7V 

2,238 (44.7) 


A ~7~7A /r~ r~ A\ 

2,770 (55.3) 




CHS 


11,217 (100.0) 


1,168 (10.4) 


10,038 (89.6) 


3,155 (28.2) 


8,046 (71.8) 


3,224 (30.6) 


7,314 (69.4) 


GGnder 


Male 


5,268 (47.0) 


514(9.8) 


4,752 (90.2) 


1,220 (23.2) 


4,046 (76.8) 


1,522 (30.6) 


3,452 (69.4) 




Female 


5,949 (53.0) 


654 (11.0) 


5,286 (89.0) 


1,935 (32.6) 


4,000 (67.4) 


1,702 (30.6) 


3,862 (69.4) 






1 , 1 [ 1 \J.o) 


70 (5.8) 


1,146(94.2) 


131 (10.8) 


1 ,085 (89.2) 


298 (26.0) 


847 (74.0) 




ou oy 


1 f!1 7 /1 R '^^ 
i ,0 1 / \ 1 O.Oj 


118(6.5) 


1,698(93.5) 


293 (16.1) 


1 ,523 (83.9) 


564 (32.5) 


1,172 (67.5) 




40-49 


9 209 n 9 R] 


183(8.3) 


2,023 (91.7) 


440 (19.9) 


1,768 (80.1) 


683 (32.2) 


1 ,440 (67.8) 




50-59 


2 07T n 8 fil 


223 (10.8) 


1,849 (89.2) 


564 (27.3) 


1,504 (72.7) 


690 (34.9) 


1,287 (65.1) 




60-69 


1 867 (1 6 8> 


238 (12.8) 


1,627 (87.2) 


777 (41.7) 


1,087 (58.3) 


571 (32.7) 


1,173 (67.3) 




Mnrp tlian 70 

LV1UI G LI lai 1 / w 


2 032 (1 8 ?] 


336 (16.5) 


1 ,695 (83.5) 


950 (46.8) 


1,079 (53.2) 


418(23.1) 


1,395 (76.9) 


Marrianp 
iviai 1 idyc 


Singie 




74 (5.3) 


1,316(94.7) 


164 (11.8) 


1,226 (88.2) 


335 (25.7) 


967 (74.3) 




Married and 


7,696 (69.3) 


om /-f n A\ 
803 (10. 4) 


c ooc /on c\ 
6,oo6 (89.6) 


2,1 bU (28.1) 


5,52/ (/1 .9) 


2,3/5 (32.0) 


A HOC /C7 n\ 

4,926 (b/.b) 




having spouse 


















Married but 


2,022 (18.2) 


285 (14.1) 


1,735(85.9) 


807 (40.0) 


1,209 (60.0) 


487 (26.5) 


1 ,351 (73.5) 




nothaving spouse 
















Education (yr) 


Less than 6 


4,263 (38.0) 


624 (14.7) 


3,634 (85.3) 


1,777 (41.8) 


2,478 (58.2) 


1,015(26.1) 


2,878 (73.9) 




7-9 


1,347 (12.0) 


150 (11.1) 


1,196(88.9) 


419(31.2) 


926 (68.8) 


402 (31 .6) 


872 (68.4) 




10-12 


3,634 (32.4) 


258 (7.1) 


3,373 (92.9) 


647 (17.8) 


2,983 (82.2) 


1,084 (31.1) 


2,399 (68.9) 




More than 13 


1,969 (17.6) 


136(6.9) 


1,831 (93.1) 


310(15.8) 


1 ,657 (84.2) 


722 (38.3) 


1,163(61 .7) 


Type of insurance 


Health insurance 


10,517(94.3) 


1,040(9.9) 


9,468 (90.1) 


2,848 (27.1) 


7,655 (72.9) 


3,057 (30.8) 


6,868 (69.2) 




Public medicai aid 


631 (5.7) 


119(20.5) 


461 (79.5) 


286 (49.3) 


294 (50.7) 


149 (28.7) 


371 (71 .3) 



Difference of vaiue is due to missing, for example 1 1 for admission, 16 for outpatient visit, and 679 for dental service in totai. HiRA, Health Insurance Review & Assessment 
Service; CHS, community health survey. 



Table 2. Sensitivity for seif-reported of healthcare utilization 



Parameters 






Admission 


Outpatient visit 




Dental visit 




No. 


Sensitivity (%) 


No. 


Sensitivity (%) 


No. 


Sensitivity (%) ^ 




Overaii 


822 


54.8 


2,133 


52.1 


1,956 


61.0 


Gender 


Male 


350 


55.8 


771 


47.8 


955 


61 .2 




Female 


472 


54.1 


1,362 


54.8 


1,001 


60.7 




P vaiue 




0.500 




< 0.001 




0.765 


Age (yr) 


19-29 


51 


38.1 


69 


30.0 


165 


51.6 




30-39 


77 


45.3 


192 


44.0 


323 


70.8 




40-49 


131 


61.8 


278 


46.1 


392 


64.6 




50-59 


155 


63.3 


370 


46.7 


436 


64.1 




60-69 


163 


54.9 


534 


58.5 


372 


60.4 




More than 70 


245 


55.4 


690 


61.5 


268 


50.6 




P vaiue 




< 0.001 




< 0.001 




< 0.001 


Marriage 


Singie 


48 


45.3 


89 


35.7 


193 


53.5 




Married and having spouse 


570 


55.1 


1,444 


50.6 


1,439 


62.6 




Married but not having spouse 


198 


56.6 


584 


61.0 


309 


59.4 




P vaiue 




0.075 




< 0.001 




0.003 


Education (yr) 


Less than 6 


445 


55.7 


1,251 


59.2 


650 


54.4 




7-9 


109 


61.9 


280 


56.3 


255 


63.1 




10-12 


171 


50.3 


410 


42.4 


646 


62.4 




More than 1 3 


97 


45.8 


191 


37.0 


405 


70.4 




P vaiue 




0.001 




< 0.001 




< 0.001 


Type of insurance 


Health insurance 


726 


54.0 


1,909 


51.2 


1,844 


61.1 




Public medicai aid 


92 


64.3 


212 


64.2 


103 


62.4 




P vaiue 




0.017 




< 0.001 




0.738 



30s (44.0%), 40s (46.1%), and 50s (46.7%), those who were sin- 
gle (35.7%) and married (50.6%), those with 10-12 (42.4%) and 
13 yr (37.0%) of education, and those who were health insurance 
holders (51.2%; P < 0.001). The sensitivity for dental visits dis- 
played no significant difference by gender, and health insurance 
type, but was significantly lower for those aged in their 20s (51.6%), 
and 70s (50.6%), those who were single (53.5%), and those who 



possessed less than 6 yr of education (54.4%; P < 0.01). 
Specificity of liealtlicare utilization 

The specificities of admission, outpatient visits, and dental vis- 
its in CHS were 96.4%, 85.6%, and 82.7%, respectively (Table 3). 
The specificity of admissions displayed no significant difference 
by gender, but showed the highest values in those who were aged 
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Table 3. Specificity for self-reported of healthcare utilization 

Admission Outpatient visit Dentai visit 
Parameters 







No. 


bpeciticity ( %) 


NO. 


bpeciticity (%) 


NO. 


bpeciticity (/o) ~ 


Overaii 




9,360 


96.4 


6,083 


85.6 


6,061 


82.7 


Gender 


Male 


4,475 


96.5 


3,205 


87.7 


3,452 


87.7 




Femaie 


4,885 


96.4 


2,878 


83.4 


3,862 


82.0 




P vaiue 




0.880 




< 0.001 




0.130 


Age (yr) 


19-29 


1,603 


98.2 


924 


93.7 


692 


83.9 




30-39 


1,605 


97.5 


1,279 


92.7 


1,039 


81.2 




40-49 


1,942 


97.4 


1,443 


89.9 


1,225 


80.8 




50-59 


1,759 


96.3 


1,082 


84.8 


1,043 


80.4 




60-69 


1,493 


95.2 


708 


74.4 


929 


82.4 




More than 70 


1,498 


94.3 


647 


71.3 


1,133 


88.3 




P vaiue 




< 0.001 




< 0.001 




< 0.001 


Marriage 


Singie 


1,258 


98.0 


1,066 


93.4 


799 


84.9 




Married and having spouse 


6,421 


96.5 


4,116 


85.2 


4,067 


81.3 




Married but not having spouse 


1,583 


94.8 


836 


78.9 


1,140 


86.5 




P vaiue 




< 0.001 




< 0.001 




< 0.001 


Education (yr) 


Less than 6 


3,308 


94.9 


1,615 


75.4 


2,333 


86.5 




7-9 


1,129 


96.5 


709 


83.6 


723 


83.1 




10-12 


3,204 


97.4 


2,425 


91.1 


2,010 


82.1 




More than 13 


1,716 


97.8 


1,332 


91.8 


993 


75.8 




P vaiue 




< 0.001 




< 0.001 




< 0.001 


Type of insurance 


Heaith insurance 


8,849 


96.6 


5,834 


86.1 


5,691 


82.4 




Pubiic medicai aid 


410 


93.8 


176 


70.4 


309 


87.0 




P value 




0.002 




< 0.001 




0.025 



in their 20s (98.2%), were single (98.0%), those who were edu- 
cated more than 13 yr (97.8%), and who were health insurance 
holders (96.6%; P < 0.01). The specificity of the outpatient visits 
showed the highest values in males (87.7%), those aged in their 
20s (93.7%), those who were single (93.4%), those who had more 
than 13 yr of education (91.8%), and those who were health in- 
surance holders (86.1%; P < 0.001). The specificity of admissions 
displayed no significant difference by gender, but showed the 
highest values in those who were aged in their 70s (88.3%), were 
married but not having spouse (86.5%), those who were educat- 
ed less than 6 yr (86.5%), and those who were public health aid 
holders (87.0%; P < 0.05). 

DISCUSSION 

The sensitivities for admission, outpatient visits, and dental visit 
were considerably lower than the specificities. Additionally, the 
sensitivity of admissions and outpatient visits were similar, de- 
spite the different utilization periods. Maybe, it seems likely that 
the respondents feel pressured to provide direct information 
regarding their medical care use, and not because they failed to 
remember. Although the sensitivities showed significant differ- 
ences according to gender, age, marital status, and education in 
individual question, there were no variables with consistent pat- 
tern on sensitivity. 

However, the sensitivity of dental visits was higher than those 
of admission and outpatient visits and the specificity was lower 
than those of admission and outpatient visits. The reason for the 
higher sensitivity appears that the respondents remembered the 



dental visits as a more specialized event than admission or out- 
patient visits. This maybe also due to the methods of obtaining 
the answers, which differed, with the admission and outpatient 
answers being only yes/no, versus the dental visits, which had 
six options for the visit period. On the conttary, reasons for the 
lower specificity appear to be the dental clinic covers more non- 
benefit items than other general medical areas, meaning the 
dental clinic cannot claim insurance benefits for all their treat- 
ments. In addition, the respondents can use all healthcare insti- 
tutions with their Industtial Accident Compensation Insurance, 
car-insurance, and non-insurance so this could lead to a higher 
false positive value for outpatient visits and dental visits. 

A questionnaire survey is one of the data sources used to iden- 
tify health behavior, health status, and healthcare utilization in 
a population, but its accuracy is known to be relatively low, when 
compared to either medical record review or insurance claim 
data (5-7). The accuracy of self-reported data in healthcare uti- 
lization depends, among other factors, on the subjects' knowl- 
edge and understanding of the relevant information, their abili- 
ty to recall information, and their willingness to report it (8). 

The range of sensitivity was 86-96% and the range of specifici- 
ty was 77%-89% with a 6-48 month follow-up period in previous 
prospective studies that investigated the validity of data on self- 
reported dental visits in the Florida Dental Care Study (9). Among 
elderly aged > 70 yr, the kappa value of hospital episodes and 
physician visits was 0.767 and 0.255, respectively, within the past 
year (10). Using the data from the 2001 Taiwan National Health 
Interview and the National Health Insurance, the crude kappa 
values for inpatients (within the past year), outpatients (within 
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the past month), and dental visits (within the past month) were 
0.75, 0.64, and 0.62, respectively (11). These values are higher 
than those reported in this study and may be explained by the 
discrepancies between the study design, questionnaire content, 
and the method of results presentation. In fact, the kappa values 
on admission, outpatients visits and dental visits in this study 
were 0.57, 0.40 and 0.44, respectively, although agreement rate 
of those were 90.9%, 73.4%, and 76.1% respectively. 

Consequently, based on these study results, increasing the sen- 
sitivity of the respondents requires effort in the process of CHS. 
First, the recall time can be subdivided into periods of 1 month, 
2-6 months, and 7 months to 1 yr for admissions, and 1 week, 2 
weeks, and 3 4 weeks for outpatient visits, and reply methods can 
be changed from healthcare institute utilization (Yes/No) to the 
utilization period, like those used for the dental visits. Indeed, it 
has been reported that an optimal recall period for self-report- 
ed surveys occurs at 6 months or less (6). Second, when health- 
care utilization questions are placed at the end of a long survey, 
the participant may want to finish the survey and not invest the 
time and effort needed for an accurate recall (12). Third, apply- 
ing the correct probing methods for the respondents is required 
when obtaining information from the past to the present. For 
example, it has been reported that a 6-2 month approach, where 
the past 6-month utilization questions are asked initially, is more 
accurate than a 2-6 month approach (13). 

The study has some limitations. First, is it right to use the in- 
surance benefit claim data of the HIRA as the gold standard for 
validity test? Although NHI did not cover aU healthcare utiliza- 
tion in Korea, NHI is a social insurance whereby aU Korean take 
out a policy for medical security as a form of health insurance 
(96.7%) or medical aid (3.3%). In addition, 89.2% of an insured 
person visited the healthcare institution as admission or outpa- 
tient more than once in 2009 (14). That is, the gap which is about 
10.8% included some people who did not use the healthcare 
institution, some people who used the healthcare institution by 
other insurance such as life-insurance, car- insurance, Industii- 
al Accident Compensation Insurance, etc or some people who 
used the healthcare institution is not covered by any insurance. 
But we did not know exact amount of coverage of other insur- 
ance or non-insurance. Anyway, considering the amount of visit 
to the healthcare institution covered by NHI and access to data 
which we used as gold standard, we thought that using the in- 
surance benefit claim data of the HIRA is the proper approach. 
Second, why did we use only 3 items, admission, outpatient visit, 
and dental visit for validity evaluation of CHS? Why did not we 
use more detailed healthcare utilization question such as out- 
patient visit by hypertension, admission by stroke, etc. In fact, 
we can evaluate the validity of specific utilization of CHS and 
compare CHS with HIRA about the accuracy of diagnosis. But 
they are the topics of the next study. In this study, we just want 
to evaluate the validity of CHS irrespective of specific disease, 



from a conservative point of view. Third, this study was a validi- 
ty test conducted only for one broad area (a self-governing dis- 
trict); as a result, we did not analyze the regional difference of 
sensitivities and specificities and the study results should not be 
generalized regarding the validity of the entire CHS. This limita- 
tion could be overcome with validity testing for aU respondents 
of the 2008 CHS (about 220,000 participants). In fact, the sensi- 
tivities and specificities of admission, outpatient visits and den- 
tal visits among 13 cities and counties of Chungbuk-do were very 
diverse. These were as follows: range of sensitivity and specific- 
ity of admission was 38.9%-67.7% and 94.1%-97.5%, range of 
sensitivity and specificity of outpatient visits was 31.2%-66.8% 
and 78.0%-92.0%, and range of sensitivity and specificity of ad- 
mission was 39.2%-74.5% and 75.6%-88.0%. It needs tiie further 
task which contains a multilevel study to explain the regional 
variation of CHS validity. 

In conclusion, despite some limitations, the present study is 
the first to evaluate the validity of nationwide health statistics 
resulting from questionnaire surveys and shows that CHS needs 
a lot of efforts to reflect the ttue health status, health behavior, 
and healthcare utilization of the population. 
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